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III. REMARKS 

Claims 1-20 are pending in the present application. In the office action dated 03/1 8/2008, 
the specification was objected to for confusing labeling of the tables. In the office action, claim 
10 was objected to for an alleged typographical error. In the office action, claims 1, 2, 4, 6, 7, 
and 10 were rejected under 35 USC § 102(b) as being anticipated by Stuart et al. (US 
20010032120). In the office action, claim 3 was rejected under 35 USC § 103(a) as being 
obvious over Stuart in view of Scarborough et al. (US 7,080,057). In the office action, claims 5, 
8, 9, and 11-13 were rejected under 35 USC § 103(a) as being obvious over Stuart in view of 
Richman (US 6,754,874). Lastly, in the office action, claim 14 was rejected under 35 USC 
§ 103(a) as being obvious over Stuart in view of Richman and in further view of Loya (US 
20020035506). Each of these items will be discussed in turn. 

A. Telephone Interview 
Applicants thank the Examiner for the courtesies extended in the telephone interview on 
May 8, 2008. During the interview, the Stuart and Richman prior art references were discussed, 
with particular focus on the failure of those references to teach or suggest using qualitative data 
to generate a performance grade. While no agreement was reached as to allowable claim 
language, Applicants appreciate the Examiner's time and comments. 

B. Specification Objections 
The specification has been amended to clearly label the tables objected to by the 
Examiner. Applicants respectfully submit that the amended specification overcomes the 
objection and request that the objection be withdrawn. 

C. Claim Objections 

Applicants respectfully note that the claim language in claim 10 is correct as originally 
submitted. The claim language refers to managing the performance of the supervisor by 
attributing the performance of the supervisor's agents to the supervisor himself. This claim as 
originally filed is supported by the original specification, drawings, and claims. See e.g., 
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Paragraphs [0031], [0040], [0043], [0079], and block 114 in Figure 2. Applicants respectfully 
submit that the Examiner's objection is overcome, and request that the objection be withdrawn. 

D, Claim Rejections Under 35 USC § 102(b) 
Claims 1, 2, 4, 6, 7, and 10 were rejected under 35 USC § 102(b) as being anticipated by 
Stuart. Stuart fails to disclose all of the limitations recited in independent claim 1 . Therefore, 
Applicants submit that the rejection under 35 USC § 102(b) should be withdrawn. 

1 . The standard for anticipation requires each and every element be found in a single reference. 
Under MPEP 2131, a claim is anticipated only if each and every element as set forth in 
the claim is found, either expressly or inherently described, in a single prior art reference. The 
identical invention must be shown in as complete detail as is contained in the claim. 

2. Stuart fails to teach generating a performance grade based in part on qualitative data. 
Applicants submit that Stuart fails to teach generating a performance grade that is at 
least in part based on qualitative data. Claim 1 of the present application requires this element, 
among others. 

The office action stated that Stuart teaches a performance grade based partially on 
qualitative data as evidenced by Paragraph [0003] of Stuart. However, Paragraph [0003] of 
Stuart merely states, as background information, that a call distributor may be able to monitor 
and report on certain qualitative aspects of an agent's performance. This statement fails to 
anticipate the requirement of claim 1 that qualitative data is used to generate a performance 
grade. This requirement of claim 1 goes beyond mere collection, monitoring, and reporting. For 
instance, the performance grade allows an agent and his supervisor to quickly and easily 
understand how the particular agent is comparing to his peers and/or to defined performance 
standards. 

Furthermore, Stuart's cost-based performance indicator is based entirely on quantitative 
data. According to the Merriam-Webster's Online Dictionary, the term "quantitative" means 
"of, relating to, or involving the measurement of quantity or amount." Whereas, the term 
"qualitative" means "of, relating to, or involving quality or kind." Merriam-Webster's Online 
Dictionary, Merriam-Webster, Inc., http://www.merriam-webster.com. These definitions are 
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consistent with these terms as used in the present application. See Paragraphs [0114], [0035], 
and [0043]. 

In Paragraph [0009], Stuart states that the cost-based performance indicator is a function 
of (1) agent call handling data and (2) agent cost data. In Paragraphs [0020]-[0021] Stuart 
describes agent call handling data as including the "number of calls received by [the] call center, 
number of calls directed toward [the] call agent team, [the] number of calls directed to each 
agent, [the] origin of incoming calls, [and the] time of day and date [ of the calls]." These 
metrics, as well as cost data, all relate to measurement of quantity or amount, and thus are forms 
of quantitative data. 

Stuart discloses a call agent performance system that feeds this data to an analysis 
module to calculate a cost-based performance indicator. See Paragraph [0026], According to 
Paragraph [0030] of Stuart, the cost-based performance indicator is cost per standard work time, 
abbreviated C/SWT where C is the cost of the agent per unit time and SWT is the standard work 
time. The agent cost per unit time, C 5 is of any desired cost metric, e.g., salary, or a combination 
of salary with other costs such as benefits and training. See Paragraph [0028]. Standard work 
time, SWT, is calculated by multiplying the work time for an agent by the percent of work time 
that the agent answered calls. See Paragraph [0032]. The cost-based performance indicator, 
C/SWT, is the quotient of the above-described cost and standard work time. Like the forms of 
collected agent call handling data discussed in the preceding paragraph, the standard work time 
relates to measurement of quantity or amount, and is thus quantitative. Therefore, it is clear from 
the above discussion that the cost-based performance indicator of Stuart, C/SWT, is a derived 
metric based entirely on quantitative data. 

In contrast, claim 1 of the application requires that qualitative data be used, at least in 
part, to generate the performance grade. In an illustrative example provided in Paragraph [0043] 
of the application, qualitative performance data is compiled from quality data and 
professionalism data. See Paragraph [0043]. Quality data involves designating the employee as 
either Key Contributor (grade level 5), Quality Plus Contributor (grade level 4), Quality 
Contributor (grade level 3), Contribution Below Expectations (grade level 2), or Contribution 
Needs Immediate Improvement (grade level 1). Paragraph [0048]. Professionalism data 
involves assessing the employee in five skill areas, e.g., Unparalleled Client Satisfaction, 
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Teamwork, Respect for the Individual, Diversity, and Integrity. Paragraph [0077], The quality 
and professionalism data relate to quality or kind, and thus are forms of qualitative data. 

In an illustrative example, in the subject application the performance grade is generated 
according to the following: 80% of the performance grade is divided among quality (based on 
overall quality score), effectiveness (based on average Average Handle Time, AHT, and After 
Call Work, ACW), and efficiency (based on schedule adherence), 10% is based on attendance 
(based on tardiness and absences), while the final 10% is professionalism (based on teamwork 
and integrity). See Paragraph [0043]. While the precise percentages of the performance grade 
may vary, it is clear that claim 1, unlike Stuart, requires qualitative data be considered in 
generating the performance grade. 

Based on the forgoing, Applicants submit that no single prior art reference of record 
discloses each and every element as set forth in present independent claim 1, as required in 
MPEP 2131. Furthermore, the prior art of record fails to teach or suggest using qualitative data 
as a component of generating a performance grade as claimed in the present application. 
Accordingly, Applicants respectfully submit that the prior art of record fails to anticipate, or 
render obvious, present claim 1. Also, where an independent claim is not anticipated or obvious, 
claims depending form the independent claim are not anticipated or obvious. Therefore, 
Applicants respectfully request that the rejection pertaining to claims 1, 2, 4, 6, 7, and 10 be 
withdrawn. 

E. Claim Rejections Under 35 USC § 103(a) 
Claim 3 was rejected under 35 USC § 103(a) as being obvious over Stuart in view of 
Scarborough. Claims 5, 8, 9, and 11-13 were rejected under 35 USC § 103(a) as being obvious 
over Stuart in view of Richman. Also, claim 14 was rejected under 35 USC § 103(a) as being 
obvious over Stuart in view of Richman and in further view of Loya. Applicants respectfully 
submit that a prima facie case of obviousness has not been established with respect to the present 
claims. Applicants note that the combined art of record fails to teach or suggest all the 
limitations required in present independent claims 1 and 11. Therefore, Applicants submit that 
the rejection under 35 USC § 103(a) should be withdrawn. 
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1 . The standard for obviousness requires that the prior art consider all claim limitations, and that 
there exists some reason to modify or combine references to arrive at the claimed invention. 
Under MPEP 2143.03, in order to establish a prima facie case of obviousness, the prior 
art reference, or combination of references, must consider all of the limitations of a claim. A 
prima facie case of obviousness also requires that there be some reason to modify or combine the 
references to arrive at the claimed invention. The mere fact that references can be combined or 
modified does not render the resultant combination obvious unless the prior art also suggests the 
results would have been predictable to one of ordinary skill in the art. MPEP 2143.01. 
Moreover, obviousness rejections "cannot be sustained by mere conclusory statements; instead, 
there must be some articulated reasoning with some rational underpinning to support the legal 
conclusion of obviousness." KSR International Co. v. Teleflex Inc., 82 USPQ2d 1385, 1395-97 
(2007) (quoting In re Kahn, 441 F.3d 977, 988 (Fed. Cir. 2006)). 

2. Combining Stuart with the art of record fails to disclose generating a performance grade 

based in part on qualitative data. 

With respect to independent claim 1, the discussion above has demonstrated that claim 1 
is novel and nonobvious over Stuart for at least the reason that claim 1 requires generating a 
performance grade based at least in part on qualitative data. Even when combining the 
teachings of the cited references, the combination still fails to disclose the use of qualitative data 
in generating a performance grade, as required by the claims of the present application. 

More specifically, Richman discloses a computer-aided evaluation system that allows 
several types of data to be captured. See Column 3, Lines 43-62. To the extent Richman 
discloses capturing qualitative data, Richman fails to disclose using such data to generate a 
performance grade. Instead, Richman merely discloses a computer-aided system designed to 
capture and store evaluative data in a convenient electronic format. See Column 6, Line 51 
through Column 11, Line 38 (describing the performance scorecard). In particular, the 
scorecards in Richman contain a listing of competency areas that further contain competency 
tasks. Column 7, Lines 37-50. Both the employee and the supervisor evaluate these competency 
tasks for the employee with a letter-rating signifying positive to negative assessments. Column 
7, Lines 51-59 and Column 9, Lines 10-15. Additionally, both the employee and the supervisor 
evaluate an overall performance rating, which is not a generated performance grade like in the 
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present claims, but instead the employee's or supervisor's overall assessment without any 
defined relationship to the specific areas and tasks evaluated. Column 8, Lines 14-34. 

Beyond Richman, neither Scarborough nor Loya disclose generating a performance grade 
based at least in part on qualitative data, within the meaning of the present claims. 

Based on the forgoing, Applicants submit that the combined art of record fails to teach or 
suggest using qualitative data as a component of generating a performance grade as claimed in 
the present application. Accordingly, Applicants respectfully submit that the combined art of 
record fails to render obvious, present claim 1 . Also, where an independent claim is not obvious, 
claims depending form the independent claim are not obvious. Therefore, Applicants 
respectfully request that the rejection pertaining to claims 3, 5, 8, and 9 be withdrawn. 

With respect to present claim 11, when combining the teachings of the cited references, 
claim 1 1 is not obvious because the combined references fail to teach or suggest at lease one 
required element of the claim. For example, for at least the reason that present claim 1 1 requires 
generating a performance score based on quality, and the combined art of record fails to disclose 
this element, claim 1 1 is not obvious under 35 USC § 103(a). 

As discussed in the above section, the present application generates a performance score 
based on quality. As used within the meaning of the present application, "quality" is a category 
of performance on similar footing as performance categories for efficiency, effectiveness, 
attendance, and professionalism. See Fig. 4. In an illustrative embodiment as discussed above, 
quality is related to an employee's level of contribution, and employees are given quality 
designations of either Key Contributor, Quality Plus Contributor, Quality Contributor, 
Contribution Below Expectations, and Contribution Needs Immediate Improvement. See 
Paragraph [0049]. 

The present office action admits on Page 20 that "Stuart/Richman do not specifically 
describe and/or disclose . . . generating a performance score based on quality . . . The office 
action continues by stating that Loya does disclose this element of claim 1 1 . However, the office 
action is incorrect in its interpretation and application of Loya as applied to the present 
application. In Loya, a point scale of 1 to 5 (1 being "unacceptable" and 5 being "excellent") is 
used to evaluate various job performance categories, such as attendance, motivation, and 
efficiency. See Paragraph [0043]. Loya, however, does not disclose a job performance category 
for quality. Despite this fact, the office action states that the 5 point value of "excellent" is 
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synonymous with generating a performance score based on quality. Here, the office action 
mischaracterizes the scale used to rate the performance categories as the performance category 
itself In the present application, quality is a distinct performance category, and generating a 
performance score based on quality means assessing an agent's performance for quality. Thus 
in the present application, it would be feasible for an agent to obtain either "unacceptable" scores 
for quality or "excellent" scores for quality. However, such scores are only measures of the 
performance category and do not constitute the performance category of quality itself Thus, 
Loya fails to describe and/or disclose the element in claim 11 requiring generating a 
performance score based on quality, within the meaning of the present claims. 

Based on the forgoing, Applicants submit that the combined art record fails to teach or 
suggest generating a performance score based on quality as claimed in the present application. 
Accordingly, Applicants respectfully submit that the combined art of record fails to render 
obvious, present claim 11. Also, where an independent claim is not obvious, claims depending 
form the independent claim are not obvious. Therefore, Applicants respectfully request that the 
rejection under 35 USC § 103(a) pertaining to present claims 11, 13, and 14 be withdrawn. 

IV. CONCLUSION 

While several distinctions have been noted over the art of record, Applicants note that 
there are several other limitations recited in the present claims that are not considered by the art 
of record. Applicants expressly reserve all rights and arguments with respect to distinctions not 
explicitly noted herein. In addition, to the extent that the amendments constitute a narrowing of 
the claims, such narrowing of the claims should not be construed as an admission as to the merits 
of the prior rejections. Applicants further note that the dependent claims include additional 
limitations not considered in the art of record, thus forming further independent bases for novelty 
and nonobviousness. 

Based on the foregoing, all pending claims are in a condition for allowance. Accordingly, 
Applicants respectfully request reconsideration and an early notice of allowance. Should the 
Examiner wish to discuss the amendments or remarks made herein, Applicants invite the 
Examiner to contact the undersigned via telephone at (513) 651-6849 or via e-mail at 
bvisconte@fbtlaw.com. 
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Respectfully Submitted, 

Barry M. Visconte (Reg. No. 52,325) 
Frost Brown Todd LLC 
2500 PNC Center 
201 East Fifth Street 
Cincinnati, Ohio 45202-4182 
513-651-6849 (direct) 
513-651-6981 (fax) 
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MARKED-UP VERSION 
PROGRAM PERFORMANCE MANAGEMENT SYSTEM 



Cross Reference to Related Applications 

[oooi] The present application hereby claims the benefit of the provisional patent 

application entitled "PROGRAM PERFORMANCE MANAGEMENT SYSTEM" to 
Shawn R. Anderson, Serial No. 60/397,651, filed on 22 July 2002. 

Field of the Invention 

[0002] The present invention relates, in general, to devices and methods that correlate 

and display employee performance evaluation factors, both objective and subjective, 
and track their updates, dissemination, and review, and more particularly to computer- 
based devices and methods particularly suited to evaluating customer service agents. 

Background of the Invention 

[0003] Accurate and timely employee evaluations are important for motivating good 

employees and taking corrective action with not-so-good employees. While this is 
generally true for all industries and services, customer service providers have a 
particular need for a comprehensive approach to agent evaluation. Each contact with 
an agent may positively or adversely impact a customer's perception of a business. 

[0004] While customer care management is a challenging service in and of itself, 

recent trends are for outsourcing this function in order to leverage customer care 
management technology, expertise, and economies of scale, However, such a 
decision is not made without reservations. For instance, a business may be concerned 
that a Customer Management Service (CMS) provider would tend to have outsourced 
agents that are not as motivated to perform their duties well as the business's own 
employees. These businesses in particular may not deem the CMS provider to have 
comprehensive and transparent program performance management capabilities to 
provide this confidence. 

[0005] Even if the CMS provider may demonstrate an agent evaluation process, a 

business may yet be concerned about how do these processes effectively manage 



performance to achieve the specific business goals of the business, rather than a 
generic, non-tailored process. Furthermore, even if tracking performance factors of 
value to the business, does the CMS provider ensure that performance feedback and 
coaching is truly delivered to agents in a timely manner to ensure its efficacy. 
Finally, even if the evaluation process is appropriate and timely for the business, 
another concern is that the performance data is unduly subjective and haphazardly 
reported. 

10006] Consequently, a significant need exists for an approach to performance 

management that is suitable for motivating agents who provide customer care, that is 
disseminated and reviewed in a timely fashion, and that is rigorously tracked and 
subject to audit to enhance confidence in its efficacy and accuracy. 

Brief Summary of the Invention 

[0007] The invention overcomes the above-noted and other deficiencies of the prior 

art by providing a performance management system and method that comprehensively 
addresses qualitative and quantitative measurands of performance for each agent and 
group of agents, intuitively displays this information in a meaningful fashion to 
various levels of supervision, including each agent, and tracks the updates, 
dissemination, and review of performance feedback through each tier of supervision. 
Sources of information are sourced and tracked in such a way that accuracy and 
objectivity are enhanced, increasing confidence. Thereby, agent performance is 
enhanced through timely and appropriate feedback. Efficacy of overall performance 
management is made transparent to each level of an organization, including a 
customer for these services. 

[0008J In one aspect of the invention, a plurality of quantitative and qualitative 

measures are selected as being aligned with appropriate business goals. These 
measures are collected, merged and analyzed in an objective manner to represent the 
various performance attributes of an agent. Results are then displayed in an intuitive 
graphical user interface that readily conveys these attributes, both individually and as 
compared to an overall group. Thereby, each agent has a current snapshot as to their 
standing in the eyes of their employer, with its implications for retention and possibly 
pay for performance, to thus motivate improved performance. Frequent reporting 



ensures that you will always know how the CMS provider and its individual agents 
are performing. Regular feedback to each agent helps ensure continuous agent 
development. 

[0009] In another aspect of the invention, a plurality of quantitative and qualitative 

measures are monitored and collected for each agent, wherein these qualitative 
measures include supervisory evaluations. Timeliness of supervisory evaluations is 
tracked, as well as agent review of feedback based on the quantitative and qualitative 
measures. 

[ooio] These and other objects and advantages of the present invention shall be made 

apparent from the accompanying drawings and the description thereof. 

Brief Description of the Figures 

[ooii] The accompanying drawings, which are incorporated in and constitute a part 

of this specification, illustrate embodiments of the invention, and, together with the 
general description of the invention given above, and the detailed description of the 
embodiments given below, serve to explain the principles of the present invention. 

[0012] FIGURE 1 is a block diagram of a Program Performance Management (PPM) 

System incorporated into a Customer Management System (CMS) network. 

[0013] FIGURE 2 is a sequence of operations performed by the PPM System of FIG. 

1. 

[0014] FIGURE 3 is a depiction of an employee scorecard graphical user interface 

(GUI) of the PPM system of FIG. 1 useful for a team leader in performing manual 
update operations and root cause analysis. 

[0015] FIGURE 4 is a depiction of an agent dashboard graphical GUI generated by 

the PPM system of FIG. 1 indicating a comparison of an agent's performance to 
standards and to peers. 

[0016] FIGURE 5 is a depiction of a queued acknowledgement form GUI generated 

by the PPM system of FIG. 1 . 
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[0017] FIGURE 6 is a depiction of recent acknowledgements report generated by the 

PPM system of FIG. 1. 

10018] FIGURE 7 is a depiction of acknowledgement detail report generated by the 

PPM system of FIG. 1. 

[0019] FIGURE 8 is a depiction of an employee performance feedback sheet 

generated by the PPM system of FIG. 1. 

[0020] FIGURE 9 is a depiction of a team leader acknowledgement queue form 

generated by the PPM system of FIG. 1 . 

[0021] FIGURE 10 is a depiction of scorecard acknowledgement event detail report 

generated by the PPM system of FIG. 1. 

[0022] FIGURE 1 1 is a depiction of an employee review rankings report generated by 

the PPM system of FIG. 1. 

[0023] FIGURE 12 is a depiction of a measure daily exclusion screen generated by 

the PPM system of FIG. 1. 

[0024] FIGURE 1 3 is a depiction of a performance trending report generated by the 

PPM system of FIG. 1. 

[0025] FIGURE 14 is a depiction of an account report generated by the PPM system 

of FIG. 1. 

[0026) FIGURE 1 5 is a depiction of an acknowledgement detail report generated by 

the PPM system of FIG. 1. 

[0027| FIGURE 16 is a depiction of an acknowledgement summary report generated 

by the PPM system of FIG. 1 . 

[0028] FIGURE 17 is a depiction of summary review form generated by the PPM 

system of FIG. 1. 

Detailed Description of the Invention 

[0029] Performance Management is the effective deployment of the right people, 

processes and technology to develop our employees for optimal results. Employees 
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who achieve outstanding business results, will earn more money, the performance 
management process ensures a consistent, standardized method in which we are 
measuring our Agents' performance and providing specific improvement opportunity 
feedback. The benefits as a result of utilizing the performance management process 
are consistency in feedback and coaching to employees across the organization; 
Employees will be able to review their status and consequently feel they have more 
control over their ratings; empowered employees, resulting in improved morale and 
job satisfaction; improved performance; and reduced attrition. 

[0030| Turning to the Drawings, wherein like numerals denote similar components 

throughout the several views, in FIG. 1, a program performance management (PPM) 
system 10 (aka "Metrex") is functionally depicted as advantageously leveraging a 
broad range of quantitative data sources available to a Consolidated Reporting 
Database (CRDB) 12 as part of a customer management system (CMS) network 14. 
In particular, The CRDB system 12 is a reporting tool utilized to access multiple 
project reports and to maintain accurate team and employee listings. The accurate 
listings are important when accessing Agent-level PPM performance data. The 
existing CRDB system 12 provides benefits include creation of reports by pulling 
from other sources, therefore eliminating the need for manual input of data; reduction 
in the time needed to pull reports and to produce reports by pulling together data from 
existing systems into one place; maintenance of accurate team and agent identification 
(IDs); and allowance for custom reporting. 

[0031] The CRDB system 12 interfaces with a number of components, processes or 

systems from which information may be received that has bearing on agent (i.e., 
employee), team leader (i.e., supervisor), project, and management performance. 
First, in an exemplary group of inputs, a Time Keeping System (TKS) 16 used for 
payroll functions. In addition to being a source of absence and tardy data on each 
agent, the TKS system 16 may detail time spent coaching, in meeting, in training, or 
administrative tasks. There may other Absentee/Tardiness Tracking components 18 
that augment what is available from a payroll-focused capability. For example, 
"clocking in" may be performed as a time and place removed from the actual worksite 
with more detailed information being available based on an agent's interaction with a 
log-in function at their station. 
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(0032] A team leader maintains a staffing/scheduling process 20, such as DIGITAL 

SOLUTIONS by , manage the schedule adherence of team members and to 

document any feedback to Agents, thereby enhancing the team statistics and 
managing the team efficiently. For absences, an agent calls an Interactive Voice 
Recognition (IVR) interface to report that he will be absent. If the Agent is absent for 
consecutive days, the Agent's file in staffing scheduling process 20 is maintained to 
adjust the number of occurrences, including adjustments for agent earnbacks and 
exceptions for approved leave of absence, such as under the Family and Medical 
Leave Act (FMLA). Other types of absence data maintained includes No Call, No 
Show (NCNS) for an entire shift as well as showing up late (i.e., tardy). 

[0033] The CRDB system 12 may advantageously interface to a Human Resources 

(HR) system 22 that provides guidelines associated with leaves of absence, 
appropriate feedback procedures, and other attendance policies. The HR system 22 
also provides updates on attrition, hiring, transfers, etc. 

[0034) The amount of time by each agent spent handling inbound calls is logged by 

an Automated Call Distribution (ACD) system 24. Similarly, the amount of time by 
each agent spent handling outbound calls is logged by Dialers 26. Sales made in 
response to an ACD call are tracked by a Sales system 28. Similarly, a wider range of 
agent contacts may be managed, such as customer contacts initiated by email or a 
website form, on a Contact Management System 30. Agents are to disposition all 
customer contacts in an Information Technology (IT) outlet so that a comparison of 
all calls handled by ACD shows that all were dispositioned. 

[0035] In addition to the range of quantitative information that represents agent 

performance, qualitative information is gathered about the agent, depicted as a quality 
system 32. One source of assessments of agent performance may be observations 
input by a team leader. Another may be by a quality assurance (QA) entity. 

[0036] These sources of information allow for the CRDB system 12 to maintain a 

range of reports: headcount reporting, attrition reporting, agent profile reporting, 
supervisory hierarchy reporting, advisor reporting, CMS ACD reporting, TKS 
reporting, IVR reporting, and Performance Management Reporting. The latter is 
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produced by the PPM system 10 in conjunction with unique PPM observations 34, 
PPM tabulations 36, and PPM review tracking 38. 

10037] The data and reporting capabilities of the CRDB system 12 and PPM system 

10 are interactively available to great advantage by administers who may customize 
the PPM system via a PPM system manual input system 40 with manual inputs 42, 
such as selecting what measures are to be assessed, weighting to be applied to the 
measures, target ranges for grading the weighted measures, and enabling inputs of 
qualitative assessments, such as comments and enhanced data capture. 

[0038| In addition, agents may access via an agent on-line review system 44 various 

non-PPM utilities 46, such as time keeping system information, schedule, paid time 
off (PTO), unpaid time off (PTO), attendance, and a Human Resources portal to 
assignment and policy guidance. On a frequent basis, the agent may access or be 
automatically provided acknowledgement feedback forms 48 as follow-up to 
supervisory feedback sessions (See FIGS. 5, 6, 7.) as well as an performance feedback 
sheet that shows trends in performance. (See FIG. 8.) In addition, the agent may 
make frequent reference to an agent dashboard 50 that comprehensively and 
intuitively depicts the agent's performance as compared to targets and as compared to 
his peers on the team. 

[0039] A team leader interacts with the PPM system 10 through a 

supervision/management computer 52 to update and monitor agent performance on an 
agent scorecard 54. When performance indications from the scorecard warrant 
corrective action, the team leader performs root cause analysis, initiates a corrective 
action plan with the agent, and inputs feedback acknowledgment tracking forms 56 
into the PPM system 10. (See FIGS. 9, 10.) The team leader or his management may 
also access PPM reports 58, such as program performance month to date, project 
scorecard status, scorecard measures applied/not applied, feedback status report, semi- 
annual performance appraisal, and semi-annual review ranking. (See FIG. 11.) 

[0040] In FIG. 2, a sequence of operations, or PPM process 1 00, is implemented by 

the PPM system 10 of FIG. 1 to effectively supervise and manage employees. It 
should be appreciated that the process 100 is depicted as a sequential series of steps 
between a team leader and an agent for clarity; however, the PPM process 100 is 



iteratively performed across an enterprise with certain steps prompted for frequent 
updates. 

[0041] In block 102, maintenance of a consolidated reporting database is performed 

so that organizational and performance related information are available, for example 
maintaining employee or agent identifiers (ID's), a supervision hierarchy, and project 
assignments, which may be more than one per employee. Typically, a team leader 
periodically reviews a listing of his direct reports maintained in a time keeping system 
to make sure that these are accurate, taking appropriate steps to initiate a change if 
warranted. 

[0042] In block 104, an administrator of the PPM system may customize what 

measures are used, the weightings given for these measures for a combined score, 
target ranges for evaluating the degree of success for each measure, implementations 
that designate how, when and by whom observations/comments are incorporated into 
the combined score, and other enhanced data capture (ENC) features. 

[0043] With the PPM system prepared, automatic performance data is compiled 

(block 106) based on Effectiveness data 108, Efficiency data 110, and Attendance 
data 1 12. These measures are rolled up as well into a similar performance tracking 
record for the team leader's performance data (block 114). In addition to quantitative 
measures, manual (qualitative) performance data is compiled (block 116) from 
Quality data 1 18 and Professionalism data 120, both typically input by the team leader 
and/or other evaluators such as QA. In the illustrative version, a scorecard has five 
categories, which total 100 points. At a high level, Quality, Effectiveness and 
Efficiency categories may be broken out in any value, but the categories must add up 
to 100%. In the illustrative version, an 80% share is divided among three categories: 
Quality (based on overall quality score), Effectiveness (based on Average Handle 
Time (AHT) and After Call Work (ACW)), and Efficiency (based on schedule 
adherence). Ten percent is Attendance (based on the tardiness and absences). The 
final ten-percent is Professionalism (based on teamwork and integrity). However, it 
should be appreciated that these specific characteristics and percentages are 
exemplary only and that other combinations may be selected for a specific application 
consistent with aspects of the invention. 
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[0044] In block 122, Managers have the ability to apply or not apply measures. This 

provides management the flexibility to compensate for elements outside an 
employee's control and correct input errors for manual measures. A "Scorecard 
Measures Apply/Not Apply report" is available to ensure that this function is used 
properly. There are a few instances when scorecard measures may need to be 
excluded from the scorecard. Some examples are shown below that illustrate when a 
measure may need to be "not applied". (See FIG. 12.) When an employee works in a 
temporary assignment that will not extend past 30 days. It may be appropriate, 
depending on the circumstances, to not apply the second scorecard's Quality and 
Efficiency measures. Note: The system automatically generates another scorecard, 
when an employee works on another team or project that has an existing scorecard. If 
a manager inputs a manual measure twice for the same month, one of the duplicate 
measures may be marked as "not applied". If something outside of employees' 
control has impacted a specific measure across the majority of the project, the 
measure may need to be not applied for the entire project. 

[0045] There are several impacts that occur when a measure is not applied. A 

measure that is "Not Applied" will not populate on the scorecard. The scorecard 
automatically changes the weightings of the scorecard, and only applied measures will 
be totaled. Not applied measures will exclude the data for that measure on higher 
level scorecards (i.e., Team Leader, Operations Manager, etc.) and all types of project 
or team level reporting. Managers will use the Metrex system to not apply or apply 
measures. The Employee Performance and Attendance folder may be selected and 
choose the "Employee Scorecard" for Agents and the "Management Scorecard" for 
Team Leaders and above. 

10046] In block 124, agent measures are calculated to determine how the agent 

compares against the standards and against their peers for the current and historical 
rating periods. 

10047] Quality score. 

[0048] A quality score is derived by pulling the overall quality score from either e- 

Talk (Advisor), Metrex Observations or EDC (Enhanced Data Capture). The final 
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score is the average of all quality evaluations for an Agent within the month. An 
exemplary formula is: 

|0049| (QA OVERALL QUALITY SCORE + TEAM LEADER OVERALL 

QUALITY SCORE) / (QA OVERALL # OF MONITORINGS + TL OVERALL # 
OF MONITORINGS) 

[0050| The above-described formula pulls automatically from either Advisor or 

Metrex Observation. If a system other than the above mentioned is utilized, manual 
entry may be necessary. In the illustrative embodiment, each measure has a set of 
five ranges that are possible to achieve, corresponding to a grade of 5, 4, 3,2, 1, and 
having the following names respectively: Key Contributor ("KC"), Quality Plus 
Contributor ("QPC"), Quality Contributor ("QC"), Contribution Below Expectations 
("CBE"), and Contribution Needs Immediate Improvement ("CNII"). Suggested 
Targets are for KC: 100% - 97%; QPC: 96% - 95%; QC: 94% - 87%; CBE: 86% - 
82%; CNII: 81% - 0. 

[oosi] Efficiency Category 

10052] Inbound Average Handle Time (AHT) is the length of time it takes for an 

Agent to handle a call. There are various factors that affect inbound AHT. The 
formula below outlines the most inclusive factors for providing the complete 
calculation for inbound AHT. An exemplary formula is: 

[0053] (IACDTIME + DAACDTIME +I_ACDAUX_OUTTIME + 

I ACDOTHERTIME + I ACWTME + I_DA_ACWTIME + TI_AUXTIME) / 
(ACDCALLS + DAACDCALLS). 

[0054] With regard to the above-described formula, the Inbound AHT calculation 

captures all three of ACD time, which includes the time an Agent spends calling out 
during a call; Hold time, which includes all of the activities an Agent performs while 
a call is on hold; and After Call Work time. The latter includes potential IB or OB 
non-ACD calls made to complete the customer's call, non-ACD calls made or 
received while in the ACW mode, and time in ACW while the Agent is not actively 
working an ACD call. 
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[0055] AUX time includes all of the AUX time captured no matter what the Agent is 

doing (i.e., including making or receiving non-ACD calls). The value of capturing all 
of the AUX time is the accountability that it creates for the Agents. It drives proper 
and accurate phone usage by Agents. 

[0056] Outbound Average Handle Time (AHT) is the length of time it takes for an 

Agent to handle a call. There are various factors that affect outbound AHT. The 
formula below outlines the most inclusive factors for providing the complete 
calculation for outbound AHT. An exemplary formula is: 

[0057] (ACW TIME + AUX OUT TIME) / (AUX OUT CALLS + AC W OUT 

CALLS) 

[0058] With regard to the above-described formula, the Outbound AHT captures the 

total time an Agent spends on a call while logged into the switch but not handling 
regular Inbound ACD calls. The ACW Time contains all of the time an Agent is in 
ACW, while logged into the phone, placing a call, and the actual Talk Time of that 
call. The AUX Out Time contains all of the time an Agent is in AUX placing calls 
and talking on calls. ACW and AUX are the only modes that Agents can place 
themselves in and still be able to place outbound calls. 

[0059] The After Call Work (ACW) percentage is the percent of time an Agent 

spends in ACW following an ACD call. It measures the percentage of actual online 
time an Agent spends in ACW without counting AUX time. This provides a clean 
view of an Agent's use of ACW to handle actual calls and removes the various 
activities that may be performed, while an Agent is in AUX. An exemplary formula 
is: 

10060] (1_ACW_T1ME + DA AC W_TIME) * 100 / (TI_STAFF_TIME - 

TI_AUX_TIME - AUX_IN_TIME - AUX_OUT_TIME) 

[006i| With regard to the above-described formula, the ACW % measure captures the 

Agent's total ACW time and calculates the percentage by dividing the total ACW time 
by the Agent's Staff time removing the Total AUX time to create a pure online time 
then multiplying by 100 to create the percentage figure. Suggested Targets are KC: 0 
- 10%; QPC: 1 1% - 15%; QC: 16% - 20%; CBE: 21% - 25%; CNII: 26% - above. 



[0062] Average After Call Work (ACW) is an actual average of the time an Agent 

spends in ACW following an ACD call. The average ACW measure provides the 
average number of seconds in ACW and is an accurate view of the actual time an 
Agent spends in ACW. For projects that bill for ACW, this measure provides a quick 
view of the potential ACW that may be included on the bill, An exemplary formula 
is: 

[0063] (LACWJTIME + DA_ACW JTIME ) / (ACDCALLS + DAACDCALLS) 

[0064] With regard to the above-described formula, Average ACW captures the 

Agent's total ACW time and calculates the average by dividing the ACW time by the 
total ACD calls the Agent receives. This provides the Agent's average, which can be 
used for projected billing when applicable. AUX time is the time an Agent spends in 
AUX work logged into the Split. True AUX time, which is the time an Agent spends 
doing various activities, provides an accurate view of the time Agents spend 
performing activities other than actual calls. An exemplary formula is: 

[0065) (TI_AUX_TIME - AUX_IN_TIME - AUX_OUT_TIME) * 1 00 / 

TISTAFFTIME 

[0066| With regard to the above-described formula, I AUX time includes I_AUX_In 

time and I_AUX_Out time. AUXIn time and AUX Out time are actually time spent 
by an Agent placing or receiving non-ACD calls, so to capture true AUX these two 
components must be removed from the total AUX time. AUX time captures all of the 
AUX reason codes to prevent Agents from selecting codes not reported. Suggested 
Targets are KC: 0 - 4%; QPC: 5% - 7%; QC: 8% - 11%; CBE: 12% - 15%; CNII: 
16% - above. 

[0067] Average Talk Time (ATT) measures the actual time spent by Agents talking to 

customers on ACD calls. This provides a clear view of the time Agents spend talking 
on calls and can be used to ensure that Agents are controlling the calls. An exemplary 
formula is: 

[0068J (ACD_TIME + DA„ACD_TIME) / (ACD_CALLS + DA_ACD_CALLS) 

[0069] With regard to the above-described formula, ATT captures the Agent's Total 

Talk time as measured in CMS (Call Management System) and divides the result by 
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the total number of ACD calls the Agent receives. It pulls the data directly from CMS 
without any components being added or removed. This makes it a pure measure of 
the Agent's actual time with the customer. 

(0070] Information Technology (IT) Sales Conversion is the percentage of sales in IT 

to ACD calls received by the Agent. This measure may contain Interlata, Intralata, or 
combined total sales. The sales type names contained in IT must be determined when 
a specific sales type conversion is desired such as Intralata conversion only. For 
example, the data label for the various sales types may be referred to as APIC rather 
than Intralata, etc. An exemplary formula is: 

[0071] (Number of Sales)* 100 / (ACD Calls) or (Number of Sales) * 100 / (IT Calls) 

10072] With regard to the above-described formula, IT Sales Conversion captures all 

sales types in IT for the project and then divides that by the total ACD Calls In or IT 
Calls, whichever is applicable, then calculates the percentage. A specific sales 
conversion can be calculated using the same calculation by selecting the appropriate 
sales type when setting up the measure in the Agent's scorecard. 

10073] The total calls dispositioned in IT vs. CMS (Call Management System) 

provides a measure to confirm whether an Agent is or is not adhering to the call 
dispositioning step in the Agent's call handling procedures. The goal should be 
around 100% to ensure that all CMS calls are being properly dispositioned in IT. An 
exemplary formula is: 

[0074] IT CALLS * 1 00 / (ACD CALLS) 

[0075] With regard to the above-described formula, the total number of calls 

dispositioned in IT divided by the total number of CMS calls received by an Agent 
then multiplied by 100. 

[0076] Effectiveness Category 

[0077] Agent Productivity is often referred to in many project as "Adjusted Agent 

Yield". This measure is intended to measure the actual online productivity of an 
Agent when handling calls. It is not an overall Billing Yield of an Agent. Therefore, 
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productive time in TKS is the only time used in this calculation. An exemplary 
formula is: 

I0078J (CMS STAFF TIME + TKS PRODUCTIVE TIME) * 1 00 / (TOTAL TKS 

TIME) 

[0079] With regard to the above-described formula, Agent Productivity captures an 

Agent's total Staff time from CMS and adds that to the Agent's actual customer 
handling productive time in TKS, which includes mail + e-mail + data entry and 
divides that total by the "clock_in seconds" or total TKS, then multiplies by 100 to 
provide a percentage format. Suggested Targets are KC: 100% - 93%; QPC: 92% - 
90%; QC: 89% - 85%; CBE: 84% - 80%; CNII: 79% - below. 

[0080] Billing Yield is used to determine the actual billable work of an Agent by 

capturing all billable time for an Agent including team meetings, training, offline non- 
customer handling time, etc. This measure is not intended to provide an Agent Yield, 
which is captured in the Agent Productivity measure. An exemplary formula is: 

[008i| (TI_STAFF TIME + (TKSJBILLABLE - TKS ONLINE) / (TKS_PAID) 

[0082] With regard to the above-described formula, Billing Yield is calculated by 

taking an Agent's Total Staff time from CMS and adds this to the Agent's total billable 
TKS time then removes the online time from TKS to avoid double counting of online 
time. This total is then divided by the Agent's total TKS. Suggested Targets are KC: 
100% - 96%; QPC: 95% - 93%; QC: 92% - 88%; CBE: 87% - 83%; CNII: 82% - 
below. 

[0083] Schedule Adherence reflects an Agent's actual adherence to their schedules 

utilized by Work Force Management. It is important to maintain accurate schedules 
in WFM and to notify the Command Center immediately of changes, as this measure 
will be negatively impacted by any change. An exemplary formula is: 

[0084] (Open In + Other In) * 1 00 / (Open In + Open Out + Other In 4- Other Out) 

[0085] Note: In other words, all of the time in adherence is divided by total scheduled 

time. With regard to the above-described formula, Schedule Adherence is calculated 
using the following data from IEX, total minutes in adherence (i.e., total number of 
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minutes the scheduled activity matches the actual activity) and compares them to the 
total minutes scheduled, then multiplies the result by 100. Suggested Targets are KC: 
100% - 95%; QPC: 94% - 93%; QC: 92% - 90%; CBE: 89% - 87%; CNII: 86% - 
below. 

[0086] Staffed to Hours Paid (HP) provides an overall view of the online Agent's 

daily time spent logged into CMS compared to the Agent's total day in TKS to 
determine whether or not the Agent is logging into the phones for the appropriate 
portion of the day. It is not intended to replace Schedule Adherence, but it provides a 
payroll view of an Agent's activities similar to Agent Productivity. An exemplary 
formula is: 

(TOTAL STAFFED TIME) MOO / (TOTAL_TK_DAY_SECONDS) 

With regard to the above-described formula, Staffed to HP captures the 
Agent's Total Staff time in CMS divided by the Agent's total TKS for the day 
multiplied by 100. Suggested Targets are KC: 100% - 90%; QPC: 89% - 87%; QC: 
86% - 82%; QBE: 81% - 77%; and CNII: 76% - below. 

Attendance is a direct feed from the Digital Solutions system (i.e., Attendance 
IVR). The feed captures occurrences, which are applied to the Agent's scorecard. 
The occurrences will only be correct when Team Leaders maintenance the Digital 
Solutions web site. Attendance is a mandatory measure and is composed of Absences 
and Tardies. Formula for Attendance is based on total number of tardies and absences 
in a calendar month. Tardies and Absences are applied directly to the automated 
scorecard from Digital Solutions. If Team Leaders do not maintenance Digital 
Solutions on a daily basis for their Agents, the Agents scorecard occurrence count will 
be inaccurate. 

[0090] The professionalism category assists Team Leaders in measuring Agents' 

performance relative to core values. There are 5 skills (i.e., Unparalleled Client 
Satisfaction, Teamwork, Respect for the Individual, Diversity, and Integrity), which 
Team Leaders manually enter into the system periodically (e.g., monthly). An 
example of a formula for professionalism is: Unparalleled Client Satisfaction (2 Pts) 
+ Teamwork (2 Pts) + Respect For The Individual (2 Pts) + Diversity (2 Pts) + 
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[0087] 
[0088] 

[0089] 



Integrity (2 Pts) = 10 Total Points Possible. These measures compose 10% of an 
Agent's scorecard. 



|009ii Team Leader Measures 

[0092| All Agent measures in the Quality, Effectiveness, and Efficiency categories 

roll up to the Team Leader's scorecard. In addition, the Team leader is evaluated for 
Attendance and Professionalism. For Attendance, a lost hours are tracked, with the 
target begin a low percentage if Team Leaders are using their scheduling system 
effectively (e.g., DIGITAL SOLUTIONS). Formula 

[0093] (IEX SCHEDULED TIME - TKS TOTAL TIME WORKED) / (IEX TOTAL 

TIME SCHEDULED) 

[0094] With regard to the above-described formula, IEX Scheduled time is the 

amount of time an Agent is scheduled to work. To alter the scheduled time, Team 
Leaders (TL) make adjustments to Digital Solutions. The adjustments are picked up 
by the Command Center and applied to their IEX Schedule. The actual TKS worked 
hours are subtracted out of the scheduled time to create the numerator. If a TL has 
maintained an Agent's schedule properly in Digital Solutions, the Lost Hours % 
should be a low number. 

[0095] Professionalism Category 

[0096] The professionalism category has been developed to assist Operations 

Managers in measuring Team Leader's performance relative to Convergys' core 
values. There are 5 skills (i.e., Unparalleled Client Satisfaction, Teamwork, Respect 
for the Individual, Diversity, and Integrity), which Operations Managers enter into the 
system, manually. An exemplary formula is: 

[0097] Unparalleled Client Satisfaction (2 pts) + Teamwork (2 pts) + Respect for the 

Individual (2 pts) + Diversity (2 pts) + Integrity (2 pts) = 10 Total Points Possible 

[0098] With regard to the above-described formula, Operations Managers input the 

manual professionalism measures monthly. These measures compose 10% of a Team 
Leader's scorecard. 
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[0099] With the cross referencing associated with the events tracked, a number of 

performance analysis tools are made available, for instance an agent scorecard 126 
that allows a team leader or manager to review the performance summaries and status 
of a number of employees. Agent trending reports 128 provides indications of 
whether a substandard performance is improving or becoming more and more of a 
problem. (See FIG. 13,) Different demographic cross sections may be selected, such 
as an account report 130 so that managers can see how particular clients of an 
outsourced service are being served by assigned employees, for instance. (See FIG. 
14). 

[ooioo] These calculations and comparisons are intuitively plotted in block 132 and 

displayed as an agent dashboard GUI 134 that gives an agent and team leader a 
frequently accessed and up-to-date snapshot of their current standing relative to the 
standards and to their peers. Also associated with these performance results are agent 
performance feedback items 136 that are created by the team leader and 
acknowledged by the agent to memorialize coaching for improved performance. 

[00101] In block 138, the team leader may reference these indications from PPM 

system in order to perform root cause analysis. Determining the root cause of any 
problem ensures that it does not reoccur in the future. Root Cause Analysis is useful 
in helping employees achieve the performance goals set by the project. A root cause 
analysis may be completed whenever an employee's performance is not meeting the 
guidelines set by the project. One technique in determining the root cause of a 
problem is to ask why three to five times, thereby eliminating the surface reasons for 
missing a target and to thus identify the root cause. The following is a list of tools 
that can help determine the root cause: Brainstorming, Cause and effect analysis 
(fishbone diagram), Histogram, Graphs, Pareto diagrams, and Checklists. Several 
steps are useful in conducting root cause analysis: (a) Enlist individuals to help in the 
root cause analysis. Include individuals that are directly affected by the outcome of 
the actions to be taken (e.g., Subject Matter Expert, another Team Leader, and/or an 
Operations Manager), (b) Conduct cause and effect analysis or use any of the helpful 
tools mentioned in this section, (c) Select the potential causes most likely to have the 
greatest impact on the problem. Note: It is not enough to identify that the root cause 
is present when the problem occurs. It must also be present when the problem does 
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not occur, (d) Create and implement an action plan to address the root causes. The 
action plan may be reviewed to ensure that the corrective actions do not cause more 
problems. 

(00102] An example of a root cause analysis checklist may be the following inquiries: 

(a) Is there a performance gap (i.e., basis, difference from target)? If so, what is the 
performance gap? Else, no further analysis required, (b) Is it worth the time and 
effort to improve (i.e., importance, cost, consequence if ignored, effect if corrected)? 
If yes, consider further its importance. No, do not waste time and effort, (c) Does the 
Team Member know that the performance is less than satisfactory (e.g., feedback 
given to team member, team member aware of unsatisfactory performance)? If yes, 
consider the basis for how you know the team member is aware that his performance 
is less than satisfactory. Else, provide appropriate feedback to the team member, (d) 
Does the Team Member know what is supposed to be done and when (i.e., objectives 
and standards been defined and mutually agreed upon and clearly stated)? If yes, how 
do you know the Team Member knows what is suppose to be done and when? Else, 
set clear goals, objectives and standards with the Team Member to clarify 
expectations, (e) Are there obstacles beyond the Team Member's control (e.g., 
conflicting demands, team member lacks necessary authority, time and/or tools, 
environmental interference such as noise or poor lighting, outdated or unduly 
restrictive policies in place)? If not, what have you done to verify that there are no 
obstacles? Else, take appropriate action to remove obstacles, (f) Are there negative 
consequences or a lack of positive consequences following positive performance, and 
in particular, how does the team member feel about the rewards for performance? If 
so, change the consequences, such as reward positive performance and work with the 
Team Member to provide appropriate support and create a developmental plan. No, 
eliminate this reason as a possibility for poor performance from the Team Member, 
(g) Are there positive consequences following non-performance? Specifically, is this 
team member receiving rewards of avoiding negative consequences even though 
performance is poor, or do they perceive other team members as doing so? What 
reward is the Team Member or other team members receiving for non- performance? 
How will you change the consequences? If yes, for instance, someone else does the 
work, if the Team Member does not do it, then change the consequences. 
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Communicate expectations to the Team Member. Create a developmental plan. Else, 
eliminate this as a possibility for poor performance from the Team Member, (h) Does 
the team member understand the consequences of poor performance? How will the 
Team Member change the performance? What will you do to provide coaching? If 
not, work with the Team Member to define consequences and create a developmental 
plan. If so, stop here. Consider lack of motivation as the problem for poor 
performance, (i) Is the Team Member willing to undertake appropriate change? If 
yes, work with the team member to create a developmental plan. If not, terminate or 
transfer the team member, or live with the performance as it is. 



TABLE 1 


Quality 






Measurement 


Review 


Action 


Call Quality 


Agent Knowledge 


Determine if changes to procedures have been 
reviewed with Agents. Determine if Agents 
understand each element of the call flow and 
the system. Determine if Agents are rushing 
through the calls. 




Expectations 


Determine if the types of improvement 
opportunities are clearly defined and 
understood by Agents. 




Other Measures 


Review the following measures to determine 
their impact on Call Quality (e.g., After Call 
Work, Average Handle Time, Attrition) 




Quality Results 


Meet with Agents to discuss trends and 
identify the root cause. 




Staffing 


Review the schedule to determine if 
appropriately staffed so that the Agent is not 
tempted to rush through the calls (i.e., look at 
staffing for peak calling periods. 


IT vs. CMS 
Call 


Call Quality 


Monitor calls and follow-up to determine if 
the calls were dispositioned correctly. 


Dispositioning 


Environment 


Observe the Agents and determine why 
Agents are not dispositioning the calls (e.g., 
talking to neighbors, etc.). Meet with Agents 
and discuss any obstacles in dispositioning 
calls correctly (e.g., coding issues). 
Determine if Agents understand the 
dispositioning procedures. 




Systems 


Determine if the codes in the system 
accurately reflect the call types. 



Tabic 1 
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TABLE 2 


Efficiency 






Measurement 


Review 


Action 


After Call 
Work (ACW) 


Observe Agent in 
Area 


Observe Agent in work environment to 
identify areas of concern, such as, excessive 
talking, searching for materials, asking 
questions, etc. 

L/Onauci siae oy siae monitoring 10 luenuiy 

improvement opportunities. 

Verify that Agents know the difference 

between ACW and AUX. 

Verify that Agents understand how to 

complete their after call work accurately. 




Floor Support 


Determine if the floor support is adequate to 
answer Agents' questions. i 




Other Measures 


Review the following measures to determine 
their impact on After Call Work: 

Average Talk Time 
Call Quality 
Occupancy 
Schedule Adherence 

oCrVlCC .LrCVCl 

Attrition 




Process / Client 
Changes 


Determine if Agents understand the new 
process/change. 

Identify if the process/change is causing 

excessive AC^W 


A TJX Time 




Ohservp A Pent*? in thp work area to nVtprminp 
if AUX Time is being used efficiently. 


r\Wl age 

Handle Time 


Review 


l\.CVICW LI1C JLUllUWlIlg lUCubUICo IU UCLCIIM1I1C 

their impact on Average Handle Time: 
After Call Work 
Average Talk Time 
Hold Time 


Average Talk 


ATT - Too High 




Time fATT) 

J. 1J11V 1 J. k 1 J. J 


ACD Time 

1 LV/J-/ X 1111 W 


Determine if this is a constant nroblem and 
review previous feedback. 




Call Quality 


Monitor at least three calls and determine if 
Agents are: 

Following the call flow correctly 
Controlling the calls 
Using reference material appropriately 
Navigating the system properly 
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TABLE 2 (Continued) 




Schedule Adherence 


Review schedule to determine if the Agent is 
following the schedule. 

Verify that the Agent is logging in and out 
properly. 




Other Measures 


Review the following measures to determine 
their impact on high Average Talk Time: 

Agent Yield 

Rill in cr Vi**1H 

Fulfillment Timeliness 

Order Comnletps ner Hnnr 

Sales per Hour 
Service Level 
Attrition 




Process 


Verify that there has not been a process 
change initiated by the client or by Convergys 
that requires a longer talk time. 




Staffing 


Review schedule to determine if staffed 
appropriately for call volume. 




Systems 


Verify that scripts are correct. 




ATT -Too Low 






Call Oualitv 


Monitor at least three calls and determine if 

iTIvlllVVl VtV 1VWU t till VV VUHJ 14- J 1 VI VlVkVl JlllliV IX 

Agents are: 

Following the call flow correctly (e.g., 
probing) 

Rushing through the call and sacrificing 
quality 




Other Measures 


Review After Call Work and Attrition to 
determine their impact on low Average Talk 
Time. 


Conversion 
Rate 


Attempts per Hour 


Compare total attempts to other Agents who 
called the same list and have a similar number 
of hours. 
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TABLE 2 (Continued) 




Call Quality 


Monitor at least three calls and determine if 
Agents are: 

Following the script / call flow 

Following the correct procedures 

to record a call as a sale 

Identifying the decision maker 

Offering services / plans and promotions that 

match customers' needs 

Overcoming objections 

Presenting benefits 

Navigating the system properly 

Recapping the calls appropriately 

Dispositioning calls correctly j 




Completed Calls 


Compare the Agent's number of completed 
calls to the team's average to determine if a 
list issue occurred. 




Other Measures 


Review the following measures to determine 
their impact on Conversion Rate: 

After Call Work 
Average Talk Time 
Dropped Calls 
Fulfillment Timeliness 
Gap Percent 

Order Completes per Hour 
Sales per Hour 
Schedule Adherence 




System 


Verify that Agent's script is correct. 




Verification Rate 


Monitor at least three calls and identify third 
party verification trends. 


Hold Time 




Monitor at least three calls to determine if: 
Hold time is used incorrectly (i.e., not using 
system or reference manual help screens, etc.). 
Customers are asking questions that Agents 
cannot answer. 

Determine if floor support is adequate. 
Determine if escalations are necessary. 


Sales Per Hour 


Attempts per Hour 


Compare total attempts to other Agents who 
called the same list and have a similar number 
of hours. 
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TABLE 2 (Continued) 




Call Quality 


Monitor at least three calls and determine if 
Agents are: 

Following the script / call flow 

Following the correct procedures 

to record a call as a sale 

Identifying the decision maker 

Offering services / plans and promotions that 

match customers' needs 

Overcoming objections 

Presenting benefits 

Navigating the system properly 

Explaining the Interlata and Intralata offers 

correctly 

Signing customers correctly for Interlata and 
Intralata offers 

Explaining third party verification process for 
Interlata and Intralata 
Recapping the calls appropriately 
Dispositioning calls correctly 




Completed Calls 


Compare the Agent's number of completed 
calls to the team's average to determine if a 
list issue occurred. 




Other Measures 


Review the following measures to determine 
their impact on Conversion Rate: After Call 
Work, Average Talk Time, Fulfillment 
Timeliness, Schedule Adherence, Order 
Completes per Hour 




Staffing (Inbound) 


Determine if the project is staffed 
appropriately to handle the calls. 




System 


Verify that Agent's script is correct. 




Verification Rate 


Monitor at least three calls and identify third 
party verification trends. 



Table 2 
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TABLE 3 


Effectiveness 






Measurement 


Review 


Action 


Agent 
Productivity 


Online Hours 


Verify the Agent was scheduled to work 
enough hours to be able to meet the goal (i.e., 
take into consideration training and vacation 
that may have been scheduled). 
Determine if off-line activities are affecting 
Agent Productivity. 

Review Agents Log In and Log Out reports to 
determine if Agents are staying online for the 
appropriate amount of time. „ 




Other Measures to 
Review 


Review the following measures to determine 
their impact on Agent Productivity: 

After Call Work 
AUX Time 
Schedule Adherence 
TKS Conformance 


Schedule 
Adherence 


Agents Changes 


Determine if the Agent's ESC and IEX 
schedule accurately reflect the Agent's 
scheduled hours. 




Environment 


Determine if Agents are following the 
attendance and tardy policy. 
Observe Agents in their work area to 
determine if Agents are talking with neighbors 
instead of logging on to the phones when 

GHTf^TV"\l'M"l P 

dppi Upi laLC. 

Review Agents Log In and Log Out reports to 
determine if Agents are staying online for the 
appropriate amount of time (i.e., leaving and 
returning from breaks on time). 




Staffing 


Determine if appropriately staffed to meet the 
volume. 




Systems 


Determine if everything is entered correctly 
into Digital Solutions. 


TKS 
Conformance 




Determine whether Agents are coding time 
appropriately in TKS. 

Meet with the Agent to determine why the 
Agent is not following TKS procedures. 



Tabic 3 
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TABLE 4 


Attendance 






Measurement 


Review 


Action 


Absenteeism/ 
Tardies 


Workplace 
Environment 


Meet with Agents to identify root cause and to 
discuss Agents' concerns. 




Schedule 


Review schedule with Agent to determine if a 
change to the schedule would eliminate further 
attendance problems. 



Table 4 



[00103] In block 140, corrective action plans are used to identify areas for 

improvement and a timeline in which expectations are to be reached. These plans 
may answer who, what, when and where and consider the conditions and approvals 
necessary for success. Action planning is used when negative trends are identified in 
an Agent's performance. Creating a plan will establish a roadmap to achieve 
excellent call quality. It also ensures an organized, objective implementation. A 
typical procedure for creating action plans would include: (1) Analyze the proposed 
improvement or solution; (2) Break it down into steps; (3) Consider the materials and 
numbers of people involved at each step; (4) Brainstorm, if necessary, for other items 
of possible significance; (5) Add to the list until the team thinks it is complete; and (6) 
Follow-up frequently to ensure the action plan is completed on time and accurately. 

[00104] In block 142, the team leader provides feedback on the agent's performance, 

including any corrective action plans that are to be implemented. Thereafter, the team 
leader captures the individual feedback items from the feedback session into the PPM 
system (block 144). Thereafter, the agent is prompted to acknowledge these items set 
into the PPM system by his team leader, perhaps with comments of explanation or 
disagreement (block 146). The PPM system tracks the setting and acknowledgement 
of feedback (block 148), which supports various reports and interactive screen to 
facilitate the process, such as acknowledgement pending/completed queues/details 
150. (See FIGS. 15, 16.) 

[00105] Periodically, the weekly or monthly or other cycle of evaluation and feedback 

is used for a review (e.g., quarterly, semi-annually, annually), which may coincide 
with compensation bonuses or raises. The PPM system tracks these periodic agent or 
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team leader review (block 152), and therefrom produces various reports or interactive 
screens to facilitate their use, such as tracking summaries 154 and an agent review 
ranking report 156. 

100106] In FIG. 3, an employee scorecard 200 allows a team leader to select one or 

more factors, such as project 202, type of employee (e.g., team leader, agent) 204, 
assigned supervisor 206 (e.g., either the team leader interacting with the screen or 
another supervisor assigned to the team leader/manager), and a period of review, such 
as start date 208 and finish date 210. Upon selecting search button 212, a listing of 
employees are provided (not depicted), typically a listing of agents assigned to the 
team leader whose log-in identifier enables him to view this subset of employees. 
One particular agent is selected with a detail employee pull-down 214, and filtering as 
desired for only applied measures yes/no radio buttons 216 and/or unreviewed 
("pending") radio button 218, with the listing displayed upon selecting a refresh 
button 220. For each performance measure, a record 222 is displayed comprised of a 
time period 224, scorecard description (e.g., project and facility) 226, scorecard 
measure 228, score 230, grade 232, apply/not apply toggle 234, and manually-entered 
comment button 236, the latter launching a text entry window for written comments. 

[00107] A top detail record 222 is shown highlighted as a currently selected record that 

may be interacted with by buttons 238-248. In particular, a "show daily detail" button 
238 will show daily statistics associated with the selected measure. A "show weekly 
detail" button 240 will show weekly statistics associated with the selected measure. A 
"show grade scale" button 242 will pop-up a legend explaining the grading scale 
standards to assist in interpreting the grades presented. A "remove scorecard for this 
employee" button 244 is used early in a feedback period to remove a pending 
scorecard until completed and restored with a "restore scorecard for this employee" 
button 246 when readily to be applied or not applied. An "add alternate project 
measures" button 248 is not grayed out when the employee is assigned to more than 
one project. Selecting button 248 allows the designating of the other projects and 
populating the scorecard with these alternate project measures. 

100108) In FIG. 4, an agent dashboard GUI 300 gives an intuitive and comprehensive 

presentation of the agent's performance as compared to standards and to his peers and 
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can be frequently referenced to instruct on areas needing attention. An individual pie 
chart 302 summarizes the 5 grade ranges by proportioning their relative weighting and 
stacking them radially from poor, fair, good, excellent and finally to outstanding. An 
arrow 304 shows the composite score for the agent, which in this instance falls within 
the good grade. A similar pie chart 306 is presented that is a summary for all of the 
team. Category measure summaries of attendance, quality, professionalism, 
efficiency and effectiveness are summarized by respective percentage values 308-316 
for both the agent and the project as well as a grade color-coded bar chart 3 1 8. 

[00i09| Performance Reports for Management Use. 

[ooiioi Senior Management Reports and Screens Guide leverages the comprehensive 

performance data and analysis of agents and team leaders to detect trends and 
problem areas. First, a Employee Performance Feedback (CRDB) report displays 
employees' (i.e., managers and Agents) month-to-date scorecard results and 
documented feedback, thereby assisting in providing coaching and feedback to 
employees. Second, an Employee Reviews (CRDB) report provides a summary of an 
employee's monthly scorecard results by category, overall points achieved and 
documented feedback, which assists in providing coaching and feedback to 
employees on their overall results. (See FIG. 17.) Third, a Program Performance 
Month-to-Date provides a roll-up of program-level scorecard data, assisting in 
managing results across teams and centers. Fourth, an Agent Productivity 
Management (APM) Cross Reference Detail Report displays the start date of a project 
and includes the following: switch number, splits, IEX code, IEX team IDs, TKS and 
PSF project code, thereby determining where the APM Measures report is pulling 
information. Fifth, an APM Measures Report with Targets and Charts sorts by 
Business Unit, center, portfolio, billing unit and PSFN project code and compares the 
following information to targets determined by the Project: agent productivity, phone 
time variance, percent occupancy, call service efficiency, percent of calls forecasted 
accurately, percent of average handle time forecasted accurately, on-line conformance 
and on-line adherence. Tracking a project's efficiency on key measures, and to 
review the accuracy of client forecasts for business planning (i.e., staffing etc.). 
Sixth, an APM Trend Report sorts by business unit, center, portfolio, billing unit and 
PSFN project code. It provides three months of project trends for the following 

-27- 



information: agent productivity, phone time variance, percent occupancy, call service 
efficiency, percent of calls forecasted accurately, percent of average handle time 
forecasted accurately, on-line conformance and on-line adherence, on-line diagnostic 
measures, breakdown of TKS categories and percent billable and non billable time. It 
identifies trends and improvement opportunities. Seventh, a TKS Activity Analysis 
Report - Detail provides project level data on where payroll time is being spent i.e., 
total coaching hours, meeting hours, training hours, etc. It assists in conducting 
project level analysis to ensure the team is following standard processes and to 
identify improvement areas. Eighth, a TKS Activity Analysis Report - Summary 
provides a summary of daily, weekly and monthly data for TKS data analysis. It also 
provides interval data and displays statistics on a project's billable time. Ninth, a 
TKS Agent Productivity Report- Detail provides by TKS Project code and employee 
the following information: manned time, other productive, productivity and phone 
time variance, thereby assisting managers in identifying how a project can be more 
efficient. Tenth, a TKS Agent Productivity Report - Summary provides by Business 
Unit, location, and TKS project(s) the following information: manned time, other 
productive, productivity and phone time variance. It assists managers in identifying 
how a project can be more efficient. Eleventh, a Yes / No Line Item Trends by Agent, 
Team Leader, and Project report provides call monitoring line item results of a 
project's evaluations completed by QA, Team Leader, OJT, client and a summary of 
all evaluations. It Assists in conducting analysis on project level quality results and 
identify areas for improvement. Twelfth, various Agent, Skill and VDN Reports 
provides a summary of monthly, weekly, daily and interval data for ACD and Agent. 
Thirteenth, Displayed project statistics report assists in managing employees' 
performance, including displaying billing data for client bills. Fourteenth, various 
ACD DN, Agent, CDN and DNIS Reports supplies a summary of monthly, weekly, 
daily and interval data for ACD, Agent, CDN and DNIS data. Displays project 
statistics. It assists in managing employees' performance and displays billing data for 
client bills. Fifteenth, various Multi-project, Project, Team Leader and Agent Level 
Reports identifies detailed and summary data daily, weekly, and monthly for key 
metrics at the following levels: multi-project, a single project, Team Leader and 
Agent. Assists in managing the key metrics and analyzing the raw data for the key 
metrics. Sixteenth, various Multi-project, Project, Team Leader and Agent Level 
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Reports identifies detailed and summary data daily, weekly, and monthly for key 
metrics at the following levels: multi-project, a single project, Team Leader and 
Agent. It assists in managing the key metrics and analyzing the raw data for the key 
metrics. Seventeenth, various Business Unit (BU), Center, Project and Job Category 
Headcount Reports supplies headcount information from CORT at the following 
levels: business unit, project and job category. This report should be pulled using the 
same start and finish date. It assists in verifying the accuracy of the information in 
CORT and developing business plans. Eighteenth, a Statistical by Interval and by 
Summary report supplies data on all calls that went through the IVR. Displays IVR 
usage, conversant routing, etc. Nineteenth, a PTO Report displays paid time off 
(PTO) information by project, team and employee, assisting in managing employee 
PTO days accrued and taken. Twentieth, various Attrition Reports (e.g., 12 Month 
Rolling, Calendar, Turnover Analysis and Employee Term Listings) supplies attrition 
information from CORT at the following levels: business unit, vertical, center, project 
and job category to assist in verifying the accuracy of the information in CORT, to 
develop action plans and strategic business plans. Twenty first, various Program, 
Finance, and Activity Summary Reports provides TKS reporting through CRDB for 
tracking and managing Agent activities, payroll, etc. 

[ooni] PPM Process Conformance is key objective of several reports that can be used 

to verify whether managers and projects are complying with the process. First, a 
Project Scorecard Status report identifies the measures that have populated on the 
scorecard. Retrieves both applied and pending measures. Identifies automated 
measures that have not populated on an individual's scorecard or need to be added 
manually. Second, a Scorecard Measures Exception Report identifies the following 
types of measures by employee name: Not Applied, Removed and Pending. It assists 
in identifying frequency of unapplied and pending measures and the manager 
responsible. Third, a Scorecards with Zero Grade displays employees who have 
received a zero due to their scores falling outside the grading criteria. It assists in 
identifying issues that need to be investigated and resolved prior to final scorecard 
processing. Fourth, a Feedback Status Report identifies, by Team Leader and Agent, 
the percent of feedback that has been acknowledged in the system. Coaching Team 
Leaders on providing timely feedback to Agents. Fifth, an Acknowledgement Detail 
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Report identifies acknowledgement type, Event number, status and by whom it was 
acknowledged by project, supervisor, and Agent. It assists in evaluating the status of 
acknowledgement types and by whom they were acknowledged. Sixth, an 
Acknowledgement Summary Report displays by business unit, center, project, 
supervisor, and Agent the following: Total number of acknowledgements, Number of 
pending acknowledgments, Percentage of completed acknowledgements, and 
Number and percentage of acknowledgements completed by a Scorecard Project 
Coordinator, Team Leader, and Agent. It assists in evaluating the completion 
percentage of acknowledgements and by whom they were acknowledged. Seventh, a 
Report Usage by Project & User Type & User identifies which employees are pulling 
reports and the reports being reviewed. It assists in providing coaching and feedback 
to managers and other employees (i.e., Reports Specialists, etc.). Eighth, a Report 
Usage by Folder BU, Report, Project Level identifies by business unit and project 
level what folders have been reviewed, thereby assessing the level of CRDB and PPM 
process usage by a project. 

[00112] Administrative - Core CRDB Agent Profile Reports identify the structure 

necessary for scorecards to accurately roll-up at each level. First, a Supervisor 
Hierarchy Report identifies the structure of a specific project, from the Agent level 
and to the President level, providing a quick and easy way to find an employee's 
manager and determine if the appropriate employees are on the list. Second, a 
Supervisor Hierarchy Audit Detail report shows by project the following employee 
information: name, Employee Number, active or inactive status, level of authority in 
CRDB, and Supervisor's Employee Number. Provides a quick view of employee 
linkages that projects can verify the accuracy of the Hierarchy report. Third, a CRDB 
CMS Dictionary provides split, VDN, and skill information at the project level and is 
utilized as a quick reference tool for managers when discussing changes with 
Workforce Management, etc. Fourth, a Project and PPM Rollup List by SME shows 
CRDB SME's by project, program, sponsor and Workforce Management group. 
Displays CRDB SME to contact when a project needs assistance, displaying agents, 
Team Leaders & Operations Managers Only. 

[00113] Some reports are used strictly by Operations Managers and Team Leaders to 

manager their employees. First, an Average Quality by Guideline and Evaluator 
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Report identifies when the first and last call monitoring evaluation was completed, 
average overall quality score and the total number of evaluations, thereby assisting in 
providing coaching and feedback to direct reports on monitoring goals and overall 
results. Second, a Quality Summary by Agent / Team Leader Report displays by 
project the Team Leaders, their Agents, number of evaluations completed per Agent, 
average overall quality score from QA, Team Leader (TL), QA & TL, OJT, client and 
all evaluations, thereby assisting in managing and providing feedback on project 
level, Team Leader level and Agent level results. Third, an Employee Review 
Rankings report ranks employees against their peers according to the points received 
on the scorecards on a monthly basis over the six-month period, determining Agent's 
appraisal ratings within a project. Fourth, a Semi-Annual Performance Appraisal 
report shows employees' performance over the six-month period, assisting in 
providing coaching and feedback to employees. Fifth, an Agent Profile by Project 
report provides Agent's name, Employee number, system ids, active status, and Team 
Leader's name, assisting Managers in troubleshooting why a measure is not 
displaying on a scorecard. Sixth, a Team Change Request Maintenance report 
provides a list of Agents and their Team Leaders by project. Transfers Agents to 
other Team Leaders within the same project, as well as transferring Agents to other 
projects. Seventh, a Manager Approval report provides Operations Managers with a 
list of pending Agent transfers; Approving or denying Agent transfer requests. 
Eighth, a Delegation of Authority report enables Operations Managers to delegate the 
authority to approve Agent transfer requests, delegating transfer approval authority 
when an Operations Manager is not in the office. Ninth, a Team Change Request 
Status Report provides a list of Team Change requests and their status, and tracks 
Team Change requests. 

In use, a program performance management (PPM) system 10 is set up as part 
of a customer management system (CMS) network 14, leveraging already existing 
quantitative information regarding employee work activities (e.g., attendance, time 
engaged in performing specific tasks, scheduling, sales results, etc.). In addition to 
automatic measures such as efficiency, effectiveness, and attendance, a team leader 
interacts with an employee scorecard 54 to input manual measures of quality and 
professionalism. These measures are compiled into a score that may be compared to 
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targets and to the peers of the employee, with the results intuitively presented to the 
employee on an agent dashboard 134. Feedback acknowledgement is facilitated by 
the PPM system 10, as well as tracking accomplishment of periodic reviews, with an 
array of reports available for upper management to evaluate agent, team leader, and 
project performance. 

[001151 While the present invention has been illustrated by description of several 

embodiments and while the illustrative embodiments have been described in 
considerable detail, it is not the intention of the applicant to restrict or in any way 
limit the scope of the appended claims to such detail. Additional advantages and 
modifications may readily appear to those skilled in the art. 

[00116] For example, although performance evaluation of agents who perform 

customer management services (CMS) is illustrated herein, it should be appreciated 
that aspects of the invention have application to other industries and services. 

[00117] What is claimed is: 
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PROGRAM PERFORMANCE MANAGEMENT SYSTEM 



Abstract of the Invention 

A Program Performance Management (PPM) system enforces consistency in feedback and 
coaching to employees across the organization lower attrition through improved morale and 
job satisfaction. Employees are empowered because they can review their status and thus 
feel that they have more control over their ratings. Consistency in performance data is 
maintained across an enterprise. Management insights are gained by comparisons made 
across projects, programs, and Business Units on standardized measures, thereby enabling 
accountability at all levels. Integration of quantitative information and qualitative 
assessments of Customer Management System (CMS) agents performance is summarized and 
plotted in an intuitive fashion, with feedback acknowledgements and reviews tracked for 
management. Team leaders have a scorecard interface to efficiently supervise their team 
members. Agents have access to a dashboard that provides up to date and intuitive 
indications of their performance and that of their fellow team members. 
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